Any Data, Any Time, Anywhere: Global Data Access for Science

Data access is key to science driven by distributed high-throughput computing (DHTC), an essential technology for many major research projects such as High Energy Physics (HEP) experiments. However, achieving efficient data access becomes quite difficult when many independent storage sites are involved because users are burdened with learning the intricacies of accessing each system and keeping careful track of data location. We present an alternate approach: the Any Data, Any Time, Anywhere infrastructure. Combining several existing software products, AAA presents a global, unified view of storage systems - a "data federation," a global filesystem for software delivery, and a workflow management system. We present how one HEP experiment, the Compact Muon Solenoid (CMS), is utilizing the AAA infrastructure and some simple performance metrics.Comment: 9 pages, 6 figures, submitted to 2nd IEEE/ACM International Symposium on Big Data Computing (BDC) 201

Software Challenges For HL-LHC Data Analysis

The high energy physics community is discussing where investment is needed to prepare software for the HL-LHC and its unprecedented challenges. The ROOT project is one of the central software players in high energy physics since decades. From its experience and expectations, the ROOT team has distilled a comprehensive set of areas that should see research and development in the context of data analysis software, for making best use of HL-LHC's physics potential. This work shows what these areas could be, why the ROOT team believes investing in them is needed, which gains are expected, and where related work is ongoing. It can serve as an indication for future research proposals and cooperations

ROOT for the HL-LHC: data format

This document discusses the state, roadmap, and risks of the foundational components of ROOT with respect to the experiments at the HL-LHC (Run 4 and beyond). As foundational components, the document considers in particular the ROOT input/output (I/O) subsystem. The current HEP I/O is based on the TFile container file format and the TTree binary event data format. The work going into the new RNTuple event data format aims at superseding TTree, to make RNTuple the production ROOT event data I/O that meets the requirements of Run 4 and beyond

Multiple-view, multiple-selection visualization of simulation geometry in CMS

Fireworks, the event-display program of CMS, was extended with an advanced geometry visualization package. ROOT's TGeo geometry is used as internal representation, shared among several geometry views. Each view is represented by a GUI list-tree widget, implemented as a flat vector to allow for fast searching, selection, and filtering by material type, node name, and shape type. Display of logical and physical volumes is supported. Color, transparency, and visibility flags can be modified for each node or for a selection of nodes. Further operations, like opening of a new view or changing of the root node, can be performed via a context menu. Node selection and graphical properties determined by the list-tree view can be visualized in any 3D graphics view of Fireworks. As each 3D view can display any number of geometry views, a user is free to combine different geometry-view selections within the same 3D view. Node-selection by proximity to a given point is possible. A visual clipping box can be set for each geometry view to limit geometry drawing into a specified region. Visualization of geometric overlaps, as detected by TGeo, is also supported. The geometry visualization package is used for detailed inspection and display of simulation geometry with or without the event data. It also serves as a tool for geometry debugging and inspection, facilitating development of geometries for CMS detector upgrades and for SLHC

Exploring server/web-client event display for CMS

The divergence of windowing systems among modern Linux distributions and OSX is making the current mode of event display operations difficult to maintain. In order to continue to support the CMS experiment event display, Fireworks, we need to explore other options beyond the current distribution model of centrally built tarballs. C++-server web-client event display is a promising direction that can maintain the full functionality of Fireworks, including operation from the full experiment framework. In addition, it brings new features like multi-user debugging and the possibility to implement more elaborate visualization of non-event data through remote access to independent services. We have been exploring mainly in the direction of Fireworks-based C++ server and thin web-client user interface as it allows for a large degree of reuse of existing algorithms as well as for full access to CMS data formats and accompanying functions that are crucial for the correct physics interpretation of event data. This paper presents the basic architecture of the system, discusses the communication protocol between server and client, and shows existing prototypes that demonstrate the feasibility of advanced event display features

EVE-7 and FireworksWeb: The next generation event visualization tools for ROOT and CMS

The CMS experiment supports and contributes to the development of the next-generation Event Visualization Environment (EVE) of the ROOT framework with the intention of superseding Fireworks, the physics analysis oriented event display of CMS, with a new server-web client implementation. EVE-7 is a rewrite of EVE for the ROOT-7 era, using modern C++ and relying on ROOT’s built-in http server for communication with GUI clients. Part of EVE-7 is also implemented in JavaScript and uses OpenUI5, JSROOT, and Three.js as its foundation libraries. While some of the advanced features of EVE have not yet been ported to EVE-7, the existing code-base can be used for building of demonstrator applications serving as technology preview. FireworksWeb is currently at the stage of a minimal application built around EVE-7. Several advanced Fireworks features have been ported into EVE-7 in an experiment-independent manner, relying heavily on Cling, the C++ interpreter of ROOT: dynamic table views, handling of physics object collections, and filtering of objects within physics collections

Exploring server/web-client event display for CMS

The divergence of windowing systems among modern Linux distributions and OSX is making the current mode of event display operations difficult to maintain. In order to continue to support the CMS experiment event display, Fireworks, we need to explore other options beyond the current distribution model of centrally built tarballs. C++-server web-client event display is a promising direction that can maintain the full functionality of Fireworks, including operation from the full experiment framework. In addition, it brings new features like multi-user debugging and the possibility to implement more elaborate visualization of non-event data through remote access to independent services. We have been exploring mainly in the direction of Fireworks-based C++ server and thin web-client user interface as it allows for a large degree of reuse of existing algorithms as well as for full access to CMS data formats and accompanying functions that are crucial for the correct physics interpretation of event data. This paper presents the basic architecture of the system, discusses the communication protocol between server and client, and shows existing prototypes that demonstrate the feasibility of advanced event display features

Controlled overflowing of data-intensive jobs from oversubscribed sites

The CMS analysis computing model was always relying on jobs running near the data, with data allocation between CMS compute centers organized at management level, based on expected needs of the CMS community. While this model provided high CPU utilization during job run times, there were times when a large fraction of CPUs at certain sites were sitting idle due to lack of demand, all while Terabytes of data were never accessed. To improve the utilization of both CPU and disks, CMS is moving toward controlled overflowing of jobs from sites that have data but are oversubscribed to others with spare CPU and network capacity, with those jobs accessing the data through real time Xrootd streaming over WAN. The major limiting factor for remote data access is the ability of the source storage system to serve such data, so the number of jobs accessing it must be carefully controlled. The CMS approach to this is to implement the overflowing by means of glideinWMS, a Condor based pilot system, and by providing the WMS with the known storage limits and let it schedule jobs within those limits. This paper presents the detailed architecture of the overflow-enabled glideinWMS system, together with operational experience of the past 6 months

HL-LHC Analysis With ROOT

ROOT is high energy physics' software for storing and mining data in a statistically sound way, to publish results with scientific graphics. It is evolving since 25 years, now providing the storage format for more than one exabyte of data; virtually all high energy physics experiments use ROOT. With another significant increase in the amount of data to be handled scheduled to arrive in 2027, ROOT is preparing for a massive upgrade of its core ingredients. As part of a review of crucial software for high energy physics, the ROOT team has documented its R&D plans for the coming years

HEP Software Foundation Community White Paper Working Group – Visualization

In modern High Energy Physics (HEP) experiments visualization of experimental data has a key role in many activities and tasks across the whole data chain: from detector development to monitoring, from event generation to reconstruction of physics objects, from detector simulation to data analysis, and all the way to outreach and education. In this paper, the definition, status, and evolution of data visualization for HEP experiments will be presented. Suggestions for the upgrade of data visualization tools and techniques in current experiments will be outlined, along with guidelines for future experiments. This paper expands on the summary content published in the HSF \emph{Roadmap} Community White Paper~\cite{HSF-CWP-2017-01